1. Field of the Invention
The invention relates generally to camera vision systems for navigation and/or object avoidance systems and, more particularly, to the processing of camera images taken through lenses that have systematic distortions.
2. Background Information
Robotic or vehicle vision navigation and/or object avoidance systems process observations made using one or more camera lenses and identify, in processed image data, various local features or objects as associated patterns in the data. The systems determine the relative locations of the features and objects, that is, locations relative to the cameras, and steer the robots or vehicles along designated paths and/or avoid objects as the robot or vehicle moves.
To process the images captured by the camera, the observations, that is, the raw image information, are processed in a known manner into pixels, or image point data. To determine relative locations of features or object represented as patterns in the processed data, the systems operate in a known manner to mathematically map the data onto a model image plane using a well-known pin-hole camera model, to determine x, y, z coordinates for the data. The coordinates are then used to determine the locations of the objects or features relative to the camera, all in a known manner.
The pin-hole camera model consists of a pin-hole prospective plane with a perspective center that is inside the physical lens, a model image plane that is a focal length f behind the lens, and an optical axis that is perpendicular to the model image plane and extends through the perspective center. Using the pin-hole camera model, light rays that pass through the lens from a point on an object located before the camera are mapped to points, and thus, coordinates on the model image plane using straight lines that pass through the perspective center. Based on the coordinates of the mapped data in the model image points the relative locations of objects or features identified as patterns in the mapped data can then be determined.
Most cameras do not conform to a “perfect” pin-hole camera in the sense that the camera lenses bend and/or distort the light rays. Accordingly, corrections are added to the image data to compensate for the departure from the pin-hole camera model. The corrections are related to what is commonly referred to as “radial symmetric lens distortion.” Examples of such distortion are barrel distortions, pin cushion distortion and combinations thereof. The corrections, which are applied to the image data as part of the mapping to the model image plane, are based on the incident angles θ of the light rays, that is, the angles of the rays from the object to the perspective center with respect to the camera optical axis.
After correction, the mapped data in the model image plane should have coordinates that would have been observed if the camera lens conformed to the perfect pin-hole camera. The locations of local features and objects relative to the camera are determined based on the corrected coordinates of the data in the associated patterns.
The larger the field of view of, for example, a fish-eye lens, the greater the distortion. The distortion associated with a fish-eye lens can be thought of as extreme radial symmetric distortion. Modeling the corrections for the fish-eye lens provides associated coordinate correction terms that grow exponentially as the incident angles increase. When the incident angles are greater than or equal to ±90°, the conventional pin-hole camera model becomes undefined, since the associated image rays that pass through the perspective center of the pin-hole plane do not intersect the model image plane. Accordingly, observations corresponding to angles of incidence greater than or equal to ±90°, i.e., information from the edges of fields of view of greater than 180°, are lost to navigation and/or object avoidance operations.
Further, errors in the corrections applied to the image data are incorporated into the measurements made using the modeled image data. For fish-eye lenses, with the correction terms becoming exponentially larger as the incident angles increase, the measurement errors also rapidly grow. Accordingly, at incident angles of approximately ±80° to ±85°, the measurements essentially become unstable. Thus, the corresponding observations are also lost to the navigation and/or object avoidance operations.
The conventional pin-hole camera model works well, even for cameras with extreme radial symmetric lens distortion, such as fish-eye lens, for many situations, because information at the edges of the field of view may not be needed. However, when the robot or vehicle is, for example, sufficiently close to a wall, the observations from the constrained field of view may consist of only data corresponding to an essentially featureless wall, and thus, be devoid of features that may aid in the navigation and/or object avoidance operations. The camera, however, through the fish-eye lens, captures other features or objects in its actual, wider, field of view, such as, for example, ceiling or floor tile patterns, over-head lighting fixtures, rug patterns, and so forth, and image data relating to these observations could be useful to the system.